• Twitter
  • LinkedIn
  • Facebook
  • Youtube
  • Rss
  • LOG IN
  • Sign up
  • FREE Trial
  • subscribe to news
Sketch Engine
  • Home
  • News & Events
    • News
    • Blog
    • Events
    • Subscribe to email updates
  • Pricing
  • Guide
    • Quick Start Guide
      • Quick Start Guide en français
    • User guide
    • Languages & corpora
    • FAQs
    • Documentation for experts
    • Training
  • About us
    • Lexical Computing
      • The company
      • The people
    • Research
      • Referencing Sketch Engine and bibliography
      • Research Agenda
      • Statistics in Sketch Engine
  • Contact
    • Contact us
    • Subscribe to email updates
  • Search
  • Menu
Blog - Latest News
You are here: Home / tokenization

tokenization

28th May 2016/in /by Michal Cukr

Tokenization is the automatic process of separating text into tokens.

Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on WhatsApp
  • Share on LinkedIn
  • Share by Mail
https://www.sketchengine.eu/wp-content/uploads/SE_logo_330x150-bleed-transp-bg.png 0 0 2016-05-28 12:17:382016-12-13 13:29:33tokenization

Connect with us!

Newsletter

This contact form is deactivated because you refused to accept Google reCaptcha service which is necessary to validate any messages sent by the form.

News

  • Display and hide statistics and counts. 5th November 2019
  • Words, tags, lemmas, lemposes, lowercase 30th October 2019
  • Parallel corpus – how to search 22nd October 2019
  • Search words and phrases in one step 16th October 2019
  • Searching for hyphenated, non-hyphenated and space-separated words in one step 8th October 2019

Archives

  • November 2019 (1)
  • October 2019 (4)
  • September 2019 (1)
  • May 2019 (1)
  • February 2019 (1)
  • January 2019 (1)
  • September 2018 (1)
  • August 2018 (2)
  • July 2018 (1)
  • June 2018 (1)
  • May 2018 (2)
  • April 2018 (2)
  • March 2018 (3)
  • February 2018 (2)
  • January 2018 (3)
  • December 2017 (3)
  • November 2017 (3)
  • October 2017 (4)
  • September 2017 (1)
  • August 2017 (3)
  • July 2017 (1)
  • June 2017 (4)
  • May 2017 (3)
  • April 2017 (2)
  • March 2017 (6)
  • February 2017 (1)
  • January 2017 (1)
  • December 2016 (5)
  • November 2016 (1)
  • October 2016 (1)
  • August 2016 (1)
  • April 2016 (2)
  • February 2016 (2)
  • January 2016 (3)
  • December 2015 (1)
  • November 2015 (2)
  • October 2015 (4)
  • September 2015 (4)
  • August 2015 (5)
  • June 2015 (4)
  • May 2015 (1)
  • April 2015 (1)
  • March 2015 (1)
  • February 2015 (1)
  • January 2015 (2)

Please enable cookie consent messages in backend to use this feature.

for learners of languages

Logo of SkELL – a tool for language learners and teachers

A Course in Lexicography and Lexical Computing

Logo of Lexicom – a workshop in corpus linguistics and lexicography
Cambridge, UK
September 2020
registration

term extraction

Logo of OneClick Terms – term extraction tool

Latest News

  • Quick TipDisplay and hide statistics and counts.5th November 2019 - 11:07 AM
  • corpus attributesWords, tags, lemmas, lemposes, lowercase30th October 2019 - 9:54 AM
  • Parallel concordanceParallel corpus – how to search22nd October 2019 - 1:00 PM
© Copyright - Lexical Computing CZ s.r.o.
  • Twitter
  • LinkedIn
  • Facebook
  • Youtube
  • Rss
  • Home
  • News & Events
  • Pricing
  • Guide
  • About us
  • Contact
token text type
Scroll to top