T
Testing / Oct 02 2019
by Simon Danisch and Andrea Amantini

Protocol

Display System

Website

  • Signup (from Nav and Pricing Page)

  • Login

  • Upgrade personal account

  • Create group

  • Upgrade group

  • editing articles on group page #4550

  • Navigating between "static" pages (logged in and logged out)

  • New notebook

  • New notebook from template

  • New private notebook

  • Import notebook

  • Deleting notebooks

  • Switching dashboards

  • Filter dashboard (all, published, drafts)

  • Contact support via Intercom

  • Billing upgrade

  • test admin privileges
  • publish article with unicode name (e.g. emojis #4477, try a name like `👋-nextjour௵alѨ` )
  • check featured articles + previews
  • from being logged out: click on nextjournal link, login, make sure to be routed to article after (broken)

Reading Notebooks

  • Remix this notebook
  • Run it
  • Transfer ownership of the notebook, e.g. to nextjournal, or any another group you're part of via settings modal, also switch visibility from public to private and back
  • Publish it
  • Follow the link that shows up on publish: it routes to the published notebook
  • Linking to published notebook with URL fragment should scroll to section (right click on outline-> open link in new tab)
  • If you enter publish modal again, it should display a Last published (with link)
  • create shared link via ("share") and make sure in incognito mode it works
  • Change MIME type for results and uploaded files, publish, make sure selected MIME sticks in published version
  • Export it as Markdown from the settings modal, re-import the file from /new, run and publish again.
  • Edit button is present on show view when you are an author
  • New Notebook button (bottom left) is present in edit mode

  • Download link of files work

  • Text and code cannot be edited in show view
  • From show view, clicking an environment in the runtimes section at the bottom of the notebook or within a code cell opens the corresponding notebook in a new tab
  • "remix of" link works
  • clicking an item in left outline menu updates the URL (in both show and edit view)

Basic Editing

  • Basic text editing with applying marks

  • insert modifier before existing text

  • Creating and editing links

  • Inline Formulas & Formulas

  • Uploading files

  • Uploading images, caption it, check display options

  • Outline

  • Restoring from history

  • Deleting nodes

  • Deleting code cells by typing backspace at beginning of cell
  • create code listings
  • name code listing

  • Deleting code listing by typing backspace at beginning of cell

  • make sure insert menu can be triggered when sidebar is collapsed
  • undo in paragraph, title, list, code cell
  • select text etc, delete, copy & paste

Code Editing

  • Adding code cells

  • adding code cell below other and having correct runtime (sometimes broken)
  • Referencing files
  • Referencing resulting file

  • Autocompletion

  • Docs Popup

  • Clicking environment link opens the corresponding notebook in new tab

  • Bash cells apply to the correct runtime

  • Inline error messages

  • Ask for help via error pop up
  • Ask for help via big `?`
  • Collapsing source code
  • Printing lots of text
  • Hiding stdout
  • Make sure hidden stdout and code cells persist in published version
  • Run via keyboard shortcut (alt + enter broken)
  • Run and focus/insert after via keyboard shortcut
  • Run via play button
  • Assigning name to cell
  • handle faulty expressions gracefully in all languages

Runtimes

  • run all for this notebook. Make sure that the output stays the same

  • verify gpu is working

  • Make sure all default images boot

  • export environment, import environment

  • changing environment variables
  • switching hardware (from started runner, from scratch) (sometimes broken)

  • switching images (from started runner, from scratch)

  • booting runner, stopping runner, stopping runner while executing, shutting down, restarting, interrupting execution, interrupting, ... (All combinations, best automated) (queuing with shutdown runner broken)
  • runner exits after shutdown (#4317)
  • mount code listing in runtime
  • read data from mounted code listing
  • mount github repo in runtime
  • read data from mounted github repo

Collaboration

  • -[ ] change runtime settings
  • -[ ] code execution / queuing (after each other, simultanously)
  • -[ ] copy paste different nodes
  • -[ ] ctrl + z
  • -[ ] go back & forth in history
  • -[ ] deleting nodes
  • -[ ] writing in same node
  • -[ ] naming code cell
  • -[ ] outline edit view
  • -[ ] move nodes
  • -[ ] adding nodes simultanously
  • -[ ] upload file
  • upload image / svg

Queuing

  • queuing cells from offline (broken)

  • queuing & interrupting

  • queuing the same cell multiple times
  • queue + create new cell

Long notebooks

  • run all
  • restore from history
  • scroll to bottom
  • export, import
  • errors in long code cell
  • outline
  • have code cells in appendix

Private accounts

  • You need a private account

  • Insert a GitHub node

  • Select "Private"

Groups

  • navigate from group secrets back to notebook (#4668)

  • publish environment from group account

runtime languages

  • add two distinct runtime-language nodes (turning on experimental mode) select 2 visibly different environments, then they should give different kernels.
  • make sure that runtime language nodes are editable and removable

  • stress test websockets

Small screens and mobile

  • run website tests + edit test

Jupyter kernel

  • test jupyter kernel (e.g. make sure bash works #4534)