Custom entities with regex pattern #86

bennyty · 2022-10-24T21:58:12Z

bennyty
Oct 24, 2022

I'm ingesting some data from Slack. Slack formats its messages with some custom templates. For example:

Custom emojis are surrounded in colons :slightly_smiling_face: and :green_checkmark:.

At-mentions are re-encoded similar to this: <@U024BE7LH> <#C024BE7LR>.

More examples and docs here: https://api.slack.com/reference/surfaces/formatting#retrieving-messages

How can I have wink tag these known regular elements? It seems like learnCustomEntities is the correct path but it needs to take a regex pattern.

Answered by sanjayaksaxena

Oct 25, 2022

Hello @bennyty

Based on the problem description, here is a possible solution:

const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );
const its = nlp.its;
// Define patterns.
const patterns = [
  { name: 'customEmoji', patterns: [ ': green _ checkmark :' ] },
  { name: 'mention', patterns: [ '< MENTION >' ] },
  { name: 'channel', patterns: [ '< HASHTAG >' ] }
];
// Train using patterns.
nlp.learnCustomEntities(patterns);
// Sample text
const text = "<@U024BE7LH> had shared a :green_checkmark: on  <#C024BE7LR> slack's channel after completing the task.";
// Read document.
const doc = nlp.readDoc( text );
// Print tokens.
con…

View full answer

sanjayaksaxena · 2022-10-25T07:28:56Z

sanjayaksaxena
Oct 25, 2022
Maintainer

Hello @bennyty

Based on the problem description, here is a possible solution:

const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );
const its = nlp.its;
// Define patterns.
const patterns = [
  { name: 'customEmoji', patterns: [ ': green _ checkmark :' ] },
  { name: 'mention', patterns: [ '< MENTION >' ] },
  { name: 'channel', patterns: [ '< HASHTAG >' ] }
];
// Train using patterns.
nlp.learnCustomEntities(patterns);
// Sample text
const text = "<@U024BE7LH> had shared a :green_checkmark: on  <#C024BE7LR> slack's channel after completing the task.";
// Read document.
const doc = nlp.readDoc( text );
// Print tokens.
console.log( doc.tokens().out() );
// Print details of each entity.
console.log( doc.customEntities().out( its.detail ) );
// Markup entities along with their type for highlighting them in the text.
doc.customEntities().each( ( e ) => {
  e.markup( '<mark>', `<sub style="font-weight:900"> ${e.out(its.type)}</sub></mark>` );
} );
// Render them as HTML!
doc.out( its.markedUpText );

The corresponding HTML output on runkit:

However you will have to keep in mind that :s is an EMOTICON (confused or worried!) and therefore the pattern for :slightly_smiling_face: would have to be written as 'EMOTICON lightly _ smiling _ face :'.

Best,
Sanjaya

2 replies

bennyty Oct 25, 2022
Author

Thank you for the detailed and extremely prompt answer! However, the reason I was asking about a regex model was that I do not know all the possible custom emojis that have been added (thousands of them) - especially if they have possible caveats such as the :s you mentioned that would make them hard or impossible to ingest automatically.

Maybe some sort of preprocessing step? I could strip them out but I'd rather not lose the information.

sanjayaksaxena Oct 25, 2022
Maintainer

Yes you are right @bennyty, since : is used in emoticons, it may be worthwhile to replace it with some character that is unlikely to occur in your context.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Custom entities with regex pattern #86

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Custom entities with regex pattern #86

Uh oh!

Uh oh!

bennyty Oct 24, 2022

Replies: 1 comment · 2 replies

Uh oh!

sanjayaksaxena Oct 25, 2022 Maintainer

Uh oh!

Uh oh!

bennyty Oct 25, 2022 Author

Uh oh!

sanjayaksaxena Oct 25, 2022 Maintainer

bennyty
Oct 24, 2022

Replies: 1 comment 2 replies

sanjayaksaxena
Oct 25, 2022
Maintainer

bennyty Oct 25, 2022
Author

sanjayaksaxena Oct 25, 2022
Maintainer