How to tokenize a string in C

how to tokenise a string in c

problem

You wish to extract tokens from a string using a delimeter. For example, a string that contains a line of code/ instruction and you wish to break it apart to further evaluate it.

SOLUTION

In C, we can find the strtok() function that helps us to break a given string into tokens using a delimeter/ separator character (e.g. a comma, tab). Below we will see an example with comma:

token.c

				
					#include <stdio.h>
#include <string.h>

int main() {

   char instruction[100] = "add $v1,$zero,$zero";
   
   //First token
   char *token = strtok(instruction, ",");

   //extract tokens
   while(token != NULL) {
      printf("%s\n", token); //printing each token
      token = strtok(NULL, ",");
   }

   return 0;
}

At line 6, we have the instruction that we wish to break into tokens using comma as separator.

At line 9, we get the first token. Then at line 12 the loop is used to extract the remaining tokens and print each one.

running

Compile:

				
					gcc token.c

Execute:

				
					./a.out

output

The instruction is split into 3 tokens as expected.

add $v1
$zero
$zero

conclusion

In this post we saw how to use the strtok() function in C to split a given string into tokens using comma as a delimeter – separator.

0 0 votes

Article Rating