February 23–26, 2020
Please log in
Please log in

Migrating AI-infused chat to Kubernetes

1:15pm2:05pm Wednesday, February 26, 2020
Location: Nassau
Secondary topics:  Best Practice, Case Study
Average rating: **...
(2.00, 2 ratings)

Who is this presentation for?

  • Software engineers, DevOps engineers, and architects




Steven Jones and Nicholas Fong explain how a team of messaging and AI engineers at IBM builds, tests, deploys, and manages 20+ customer-facing chatbots that serve people across various geographies and languages on the web every day.

The chatbots had to be able to answer customers’ most common questions, navigate users to the right resource, and be able to transfer the chat to the most knowledgeable human. To process natural language queries, IBM needed a Python-based microservice that could leverage that vast number of natural language and data processing libraries already available in Python. This service leveraged large existing language models as well as custom-built models and had high disk, CPU, and memory requirements. IBM fronted the Python-based microservice with a Node.js Express server to be able to handle large request volume, hand-off to other microservices, or to its own NLP microservice. The Express server also services IBM’s frontend chat experience. These same JavaScript developers worked on this microservice. This Node.js microservice was lean and required little memory and disk space.

The teams needed an easy way to manage, deploy, scale, and update these microservices. They found out-of-the-box platform-as-a-service (PaaS) technologies had CPU, memory, and disk limitations that prevented them from leveraging large models. Bare-metal servers are hard to manage. IBM’s microservices intercommunicate frequently, so network communication speed is important. Kubernetes met all of the teams’ needs: it’s scalable, flexible, manageable, and fast.

Steven and Nicholas walk you through migrating IBM’s chatbot, cognitive search, and other services to a Kubernetes-based architecture. Technologies include multiregion clusters, load balancers, integrating Express and Flask servers, and high-speed data transfer for importing models.

Prerequisite knowledge

  • A basic understanding of software development and cloud computing

What you'll learn

  • Discover when to use a PaaS versus Kubernetes for managing microservices
  • Learn about hosting machine learning models in Kubernetes
Photo of Steven Jones

Steven Jones


Steven W Jones is the lead architect of messaging and AI for the inbound marketing channel on IBM.com, cloud platform, and in-app infusing AI into chat, messaging, scheduler, chatbots, and transcript analysis.

Photo of Nicholas Fong

Nicholas Fong


Nicholas Fong is a full stack engineer working at IBM. When not working on delighting users, he enjoys exploring New York City for new restaurants and playing with his cat, Miko.

  • IBM
  • LaunchDarkly
  • LightStep
  • Red Hat
  • ThoughtWorks
  • Auth0
  • Check Point Software
  • Contentful
  • Contrast Security
  • Datadog
  • Diamanti
  • Octobot.io
  • Optimizely
  • Perforce
  • Robin.io
  • SmartBear
  • Tidelift
  • WhiteSource
  • Synopsys
  • AxonIQ
  • Codefresh
  • CodeStream
  • Hello2morrow
  • LogRocket
  • Rookout
  • Solo.io
  • CNN
  • Boundless Notions, LLC

Contact us


For conference registration information and customer service


For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference


For media/analyst press inquires