文本分辨认知技能

文本分割技能将文本拆分成块文字。 你可以指定是想将文本拆分成句子，还是按特定长度的页面。 位置元数据如偏移和序数位置也可作为输出提供。 如果下游技能有最大文本长度要求，比如将数据块传递到Azure OpenAI及其他模型提供者嵌入模型的嵌入技能，这项技能非常有用。 有关该情景的更多信息，请参见 向量搜索的块文件。

若干参数是版本特定的。 技能参数表会注明引入参数的API版本，方便你判断是否需要 升级版本 。 要使用2024-09-01-preview中使用token分区等版本特定功能，您可以使用Azure门户，或针对REST API版本，或查看Azure SDK更改日志是否支持该功能。

Azure门户支持大多数预览功能，并可用于创建或更新技能集。 关于文本拆分技能的更新，请编辑技能集JSON定义以添加新的预览参数。

@odata.type

Microsoft.Skills.Text.SplitSkill

Skill Parameters

参数区分大小写。

Skill Inputs

Skill Outputs

Sample definition

{
    "name": "SplitSkill", 
    "@odata.type": "#Microsoft.Skills.Text.SplitSkill", 
    "description": "A skill that splits text into chunks", 
    "context": "/document", 
    "defaultLanguageCode": "en", 
    "textSplitMode": "pages", 
    "unit": "azureOpenAITokens", 
    "azureOpenAITokenizerParameters":{ 
        "encoderModelName":"cl100k_base", 
        "allowedSpecialTokens": [ 
            "[START]", 
            "[END]" 
        ] 
    },
    "maximumPageLength": 512,
    "inputs": [
        {
            "name": "text",
            "source": "/document/text"
        },
        {
            "name": "languageCode",
            "source": "/document/language"
        }
    ],
    "outputs": [
        {
            "name": "textItems",
            "targetName": "pages"
        }
    ]
}

Sample input

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "text": "This is the loan application for Joe Romero, a Microsoft employee who was born in Chile and who then moved to Australia...",
                "languageCode": "en"
            }
        },
        {
            "recordId": "2",
            "data": {
                "text": "This is the second document, which will be broken into several pages...",
                "languageCode": "en"
            }
        }
    ]
}

Sample output

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "pages": [
                    "This is the loan...",
                    "In the next section, we continue..."
                ],
                "offsets": [
                    {
                        "utf8": 0,
                        "utf16": 0,
                        "codePoint": 0
                    },
                    {
                        "utf8": 146,
                        "utf16": 146,
                        "codePoint": 146
                    }
                ],
                "lengths": [
                    {
                        "utf8": 146,
                        "utf16": 146,
                        "codePoint": 146
                    },
                    {
                        "utf8": 211,
                        "utf16": 211,
                        "codePoint": 211
                    }
                ],
                "ordinalPositions" : [
                    1,
                    2
                ]
            }
        },
        {
            "recordId": "2",
            "data": {
                "pages": [
                    "This is the second document...",
                    "In the next section of the second doc..."
                ],
                "offsets": [
                    {
                        "utf8": 0,
                        "utf16": 0,
                        "codePoint": 0
                    },
                    {
                        "utf8": 115,
                        "utf16": 115,
                        "codePoint": 115
                    }
                ],
                "lengths": [
                    {
                        "utf8": 115,
                        "utf16": 115,
                        "codePoint": 115
                    },
                    {
                        "utf8": 209,
                        "utf16": 209,
                        "codePoint": 209
                    }
                ],
                 "ordinalPositions" : [
                    1,
                    2
                ]
            }
        }
    ]
}

分块与矢量化示例

这个例子是针对积分向量化的。

• pageOverlapLength：重叠文本在 数据分块 场景中非常有用，因为它保持了从同一文档生成的块之间的连续性。

• maximumPagesToTake：页数限制在矢 量化 场景中非常有用，因为它帮助你控制在提供矢量化的嵌入模型的最大输入限制以下。

Sample definition

该定义加上 pageOverlapLength 100个字符和 maximumPagesToTake 1个字符。

假设 maximumPageLength 是 5,000 字符（默认），那么 "maximumPagesToTake": 1 处理每个源文档的前 5,000 字符。

该示例设 textItems 为 至 myPagestargetName。 因为 targetName 是设置的， myPages 你应该用这个值来选择文本分割技能的输出。 在下游技能中，索引器输出/document/myPages/*字段映射、知识库预测和索引预测。

{
    "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
    "textSplitMode" : "pages", 
    "maximumPageLength": 1000,
    "pageOverlapLength": 100,
    "maximumPagesToTake": 1,
    "defaultLanguageCode": "en",
    "inputs": [
        {
            "name": "text",
            "source": "/document/content"
        },
        {
            "name": "languageCode",
            "source": "/document/language"
        }
    ],
    "outputs": [
        {
            "name": "textItems",
            "targetName": "myPages"
        }
    ]
}

采样输入（与前例相同）

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "text": "This is the loan application for Joe Romero, a Microsoft employee who was born in Chile and who then moved to Australia...",
                "languageCode": "en"
            }
        },
        {
            "recordId": "2",
            "data": {
                "text": "This is the second document, which will be broken into several sections...",
                "languageCode": "en"
            }
        }
    ]
}

采样输出（注意重叠）

在每个“textItems”数组中，第一个条目后面的文本会被复制到第二个条目的开头。

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "myPages": [
                    "This is the loan...Here is the overlap part",
                    "Here is the overlap part...In the next section, we continue..."
                ]
            }
        },
        {
            "recordId": "2",
            "data": {
                "myPages": [
                    "This is the second document...Here is the overlap part...",
                    "Here is the overlap part...In the next section of the second doc..."
                ]
            }
        }
    ]
}

Error cases

如果某语言不被支持，会生成警告。

See also

• Built-in skills
• 如何定义技能组