how to tokenise a string in c

problem

You wish to extract tokens from a string using a delimeter. For example, a string that contains a line of code/ instruction and you wish to break it apart to further evaluate it.

SOLUTION

In C, we can find the strtok() function that helps us to break a given string into tokens using a delimeter/ separator character (e.g. a comma, tab). Below we will see an example with comma:

token.c
				
					#include <stdio.h>
#include <string.h>

int main() {

   char instruction[100] = "add $v1,$zero,$zero";
   
   //First token
   char *token = strtok(instruction, ",");

   //extract tokens
   while(token != NULL) {
      printf("%s\n", token); //printing each token
      token = strtok(NULL, ",");
   }

   return 0;
}
				
			

At line 6, we have the instruction that we wish to break into tokens using comma as separator.

At line 9, we get the first token. Then at line 12 the loop is used to extract the remaining tokens and print each one.

running

Compile:

				
					gcc token.c
				
			

Execute:

				
					./a.out
				
			
output

The instruction is split into 3 tokens as expected.

  • add $v1
  • $zero
  • $zero

conclusion

In this post we saw how to use the strtok() function in C to split a given string into tokens using comma as a delimeter – separator.

Facebook
Twitter
LinkedIn
Reddit
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x