• Twitter
  • LinkedIn
  • Facebook
  • Youtube
  • Rss
  • LOG IN
  • Sign up
  • FREE Trial
  • subscribe to news
Sketch Engine
  • Home
  • News & Events
    • News
    • Blog
    • Events
    • Subscribe to email updates
  • Pricing
  • Guide
    • Quick Start Guide
      • Quick Start Guide en français
    • User guide
    • Languages & corpora
    • FAQs
    • Documentation for experts
    • Training
  • About us
    • Jobs
    • Lexical Computing
      • The company
      • The people
    • Research
      • Referencing Sketch Engine and bibliography
      • Research Agenda
      • Dissertation topics
      • Statistics in Sketch Engine
  • Contact
    • Contact us
    • Subscribe to email updates
  • Search
  • Menu Menu
Blog - Latest News
You are here: Home1 / tokenization

tokenization

28th May 2016/in /by Michal Cukr

Tokenization is the automatic process of separating text into tokens. This process is performed by tools called tokenizers.

Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on WhatsApp
  • Share on LinkedIn
  • Share by Mail
https://www.sketchengine.eu/wp-content/uploads/SE_logo_330x150-bleed-transp-bg.png 0 0 2016-05-28 12:17:382023-05-29 12:27:47tokenization

Connect with us!

Newsletter

News

  • Sketch Engine news: enhanced tools, new corpora, and Lexicom 2024 in Spain 1st November 2023
  • Boot Camp – a face-to-face course in using Sketch Engine 5th October 2023
  • No “business as usual” with Russia anymore 7th March 2022
  • More concordance context 18th March 2020
  • Spoken British National Corpus 2014 18th February 2020

Archives

  • November 2023 (1)
  • October 2023 (1)
  • March 2022 (1)
  • March 2020 (1)
  • February 2020 (1)
  • January 2020 (2)
  • December 2019 (1)
  • November 2019 (1)
  • October 2019 (4)
  • September 2019 (1)
  • May 2019 (1)
  • February 2019 (1)
  • January 2019 (1)
  • September 2018 (1)
  • August 2018 (2)
  • July 2018 (1)
  • June 2018 (1)
  • May 2018 (2)
  • April 2018 (2)
  • March 2018 (3)
  • February 2018 (2)
  • January 2018 (3)
  • December 2017 (3)
  • November 2017 (3)
  • October 2017 (4)
  • September 2017 (1)
  • August 2017 (3)
  • July 2017 (1)
  • June 2017 (4)
  • May 2017 (3)
  • April 2017 (2)
  • March 2017 (6)
  • February 2017 (1)
  • January 2017 (1)
  • December 2016 (5)
  • November 2016 (1)
  • October 2016 (1)
  • August 2016 (1)
  • April 2016 (2)
  • February 2016 (2)
  • January 2016 (3)
  • December 2015 (1)
  • November 2015 (2)
  • October 2015 (4)
  • September 2015 (4)
  • August 2015 (4)
  • June 2015 (4)
  • May 2015 (1)
  • April 2015 (1)
  • March 2015 (1)
  • February 2015 (1)
  • January 2015 (2)

for learners of languages

SkELL logo transparent

A Course in Lexicography and Lexical Computing

Logo of Lexicom – a workshop in corpus linguistics and lexicography

term extraction

Logo of OneClick Terms – term extraction tool

learn sketch engine

© Copyright - Lexical Computing CZ s.r.o.
  • Twitter
  • LinkedIn
  • Facebook
  • Youtube
  • Rss
  • Contact
  • Jobs
  • Pricing
  • Privacy
  • Terms
  • Cookie settings
tokentext type
Scroll to top