• Twitter
  • Linkedin
  • Facebook
  • Youtube
  • Gplus
  • Rss
  • Register
  • FREE Trial
  • subscribe to news
Sketch Engine
  • Home
  • News & Events
    • News
    • Blog
    • Events
    • Subscribe to email updates
  • Pricing
  • Guide
    • Quick Start Guide
      • Quick Start Guide en français
    • User guide
    • Languages & corpora
    • FAQs
    • Documentation for experts
  • About us
    • Lexical Computing
      • The company
      • The people
    • Research
      • Referencing Sketch Engine and bibliography
      • Research Agenda
      • Statistics in Sketch Engine
  • Contact
    • Contact us
    • Subscribe to email updates
  • Search
  • Menu
Blog - Latest News
You are here: Home / tokenization

tokenization

28th May 2016/in /by Michal Cukr

Tokenization is the automatic process of separating text into tokens.

Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on Google+
  • Share on Linkedin
  • Share by Mail
https://www.sketchengine.eu/wp-content/uploads/SE_logo_330x150-bleed-transp-bg.png 0 0 2016-05-28 12:17:382016-12-13 13:29:33tokenization

Connect with us!

Newsletter

News

  • Build a corpus from the web 13th May 2019
  • Sketch Engine calendar 2019 – March 28th February 2019
  • Sketch Engine calendar 2019 – January 10th January 2019
  • October calendar page with word sketches 26th September 2018
  • Your data are safe with us. Now officially! 28th August 2018

Archives

  • May 2019 (1)
  • February 2019 (1)
  • January 2019 (1)
  • September 2018 (1)
  • August 2018 (2)
  • July 2018 (1)
  • June 2018 (1)
  • May 2018 (2)
  • April 2018 (2)
  • March 2018 (3)
  • February 2018 (2)
  • January 2018 (3)
  • December 2017 (3)
  • November 2017 (3)
  • October 2017 (4)
  • September 2017 (1)
  • August 2017 (3)
  • July 2017 (1)
  • June 2017 (4)
  • May 2017 (3)
  • April 2017 (2)
  • March 2017 (6)
  • February 2017 (2)
  • January 2017 (1)
  • December 2016 (5)
  • November 2016 (1)
  • October 2016 (1)
  • August 2016 (1)
  • April 2016 (2)
  • February 2016 (2)
  • January 2016 (3)
  • December 2015 (1)
  • November 2015 (2)
  • October 2015 (4)
  • September 2015 (4)
  • August 2015 (5)
  • June 2015 (4)
  • May 2015 (1)
  • April 2015 (1)
  • March 2015 (1)
  • February 2015 (1)
  • January 2015 (2)

for learners of languages

Logo of SkELL – a tool for language learners and teachers

A Course in Lexicography and Lexical Computing

Logo of Lexicom – a workshop in corpus linguistics and lexicography
Mikulov, CZ, June 2019
registration

term extraction

Logo of OneClick Terms – term extraction tool

Latest News

  • corpus from the webBuild a corpus from the web13th May 2019 - 3:12 PM
  • Sketch Engine calendar 2019Sketch Engine calendar 2019 – March28th February 2019 - 5:28 PM
  • Sketch Engine calendar 2019Sketch Engine calendar 2019 – January10th January 2019 - 4:47 PM
© Copyright - Lexical Computing CZ s.r.o.
  • Twitter
  • Linkedin
  • Facebook
  • Youtube
  • Gplus
  • Rss
  • Home
  • News & Events
  • Pricing
  • Guide
  • About us
  • Contact
token text type
Scroll to top