• Twitter
  • Linkedin
  • Facebook
  • Youtube
  • Gplus
  • Rss
  • Register
  • FREE Trial
  • subscribe to news
Sketch Engine
  • Home
  • News & Events
    • News
    • Blog
    • Events
    • Subscribe to email updates
  • Get Sketch Engine
  • User Guide
    • Quick Start Guide
      • Quick Start Guide en français
    • User guide
    • Corpora
    • Glossary
    • Documentation
  • FAQ
  • About us
    • Lexical Computing
      • The company
      • The people
    • Research
      • Referencing Sketch Engine and bibliography
      • Research Agenda
      • Statistics in Sketch Engine
  • Contact
    • Contact us
    • Subscribe to email updates
  • Search
  • Menu
Blog - Latest News
You are here: Home / tokenization

tokenization

28th May 2016/in /by Michal Cukr

Tokenization is the automatic process of separating text into tokens.

Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on Google+
  • Share on Linkedin
  • Share by Mail
https://www.sketchengine.eu/wp-content/uploads/SE_logo_330x150-bleed-transp-bg.png 0 0 2016-05-28 12:17:382016-12-13 13:29:33tokenization

Connect with us!

Newsletter

News

  • October calendar page with word sketches 26th Sep 2018
  • Your data are safe with us. Now officially! 28th Aug 2018
  • Sketch Engine calendar 2018 – September 23rd Aug 2018
  • Sketch Engine calendar 2018 – August 26th Jul 2018
  • Sketch Engine calendar 2018 – July 28th Jun 2018

Archives

  • Sep 2018 (1)
  • Aug 2018 (2)
  • Jul 2018 (1)
  • Jun 2018 (1)
  • May 2018 (2)
  • Apr 2018 (2)
  • Mar 2018 (3)
  • Feb 2018 (2)
  • Jan 2018 (3)
  • Dec 2017 (3)
  • Nov 2017 (3)
  • Oct 2017 (4)
  • Sep 2017 (1)
  • Aug 2017 (3)
  • Jul 2017 (1)
  • Jun 2017 (4)
  • May 2017 (3)
  • Apr 2017 (2)
  • Mar 2017 (6)
  • Feb 2017 (2)
  • Jan 2017 (1)
  • Dec 2016 (5)
  • Nov 2016 (1)
  • Oct 2016 (1)
  • Aug 2016 (1)
  • Apr 2016 (2)
  • Feb 2016 (2)
  • Jan 2016 (3)
  • Dec 2015 (1)
  • Nov 2015 (2)
  • Oct 2015 (4)
  • Sep 2015 (4)
  • Aug 2015 (5)
  • Jun 2015 (4)
  • May 2015 (1)
  • Apr 2015 (1)
  • Mar 2015 (1)
  • Feb 2015 (1)
  • Jan 2015 (2)

for learners of languages

Logo of SkELL – a tool for language learners and teachers

A Course in Lexicography and Lexical Computing

Logo of Lexicom – a workshop in corpus linguistics and lexicography
Cambridge, UK, September 2018
registration

term extraction

Logo of OneClick Terms – term extraction tool

Latest News

  • Sketch Engine calendar 2018October calendar page with word sketches26th Sep 2018 - 1:57 PM
  • Your data are safe with us. Now officially!28th Aug 2018 - 4:54 PM
  • Sketch Engine calendar 2018Sketch Engine calendar 2018 – September23rd Aug 2018 - 1:57 PM
© Copyright - Lexical Computing CZ s.r.o.
  • Twitter
  • Linkedin
  • Facebook
  • Youtube
  • Gplus
  • Rss
  • Home
  • News & Events
  • Get Sketch Engine
  • User Guide
  • FAQ
  • About us
  • Contact
token text type
Scroll to top