Developer PSA: labs.j-novel.club is your new friend!

chocolatkey

@redmasq There's really no point in restricting it in that manner, which can simply be circumvented by using a non-browser client or setting up a proxy to the API. More work for me, more work for you, zero sum game

redmasq

@chocolatkey I only ask since such extra measures have been en vogue as of late. I was thinking of doing a plugin for my copy of Calibre to get updates to ebooks and such.

chocolatkey

Added https://labs.j-novel.club/app/v1/auth/login and https://labs.j-novel.club/app/v1/auth/logout

Login payload (in JSON) is {"login":"<email or username>","password":"<password>"}. Optionally, slim: true can be added if all you need is a token, and don't need cookies.
400 is your fault, 404 means user not found, 401 means bad pass, 429 means too many tries, else assume a temporary outage or bug

Logout is just POST with valid auth to delete session associated with token and expire the cookies.

Icerius

@chocolatkey I would recommend returning 401 for both user not found and invalid password.

It is considered best practice to return the same result during authentication irrespective of if the error is invalid user or invalid pass

See chapter "Authentication and Error messages" in https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html for more info

chocolatkey

@icerius that's a fair point, it's changed now. There are still technically ways to enumerate users, thought I won't mention them.

redmasq

@chocolatkey Thanks for the hard work.

Concerning the username scraping, with that vector plugged up, the easiest way, if reading above correctly, would be once logged in, using rainbow tables, but a moot point since I'm fairly certain someone with a valid username can harvest from these forums anyways, in theory. Logging mixed with governor limits per login identity and per IP/subnet should mitigate much of that style. At least for stuff that I have maintained, an expiring minutely and a rolling daily counter set usually suffices. That selection allowed me to manage the minutely collection in-memory and queue up daily to allow atomic transactions for log and aggregate counts.

Regardless, hoping to setup keeping my local copies auto synced, so might code up a cron job this weekend. I may be willing to share a wrapper of whatever language I use if there is interest. Node.js (JavaScript), Python, or Powershell will be my most likely one of my choices, provided I don't get assigned honey-dos (not that this sort of thing is much more than http request and deserialization, but wouldn't be the first time).

chocolatkey

This post is deleted!

Crimson Wise

I am looking for some dogfooding

For reference, which kind of feedback are you looking for?

jhkghl678690iop

One thing is in order to get all the parts of the serie, it is possible to call:

GET https://labs.j-novel.club/app/v1/parts/{ID or slug}/toc

But the ID of a part of the series needs to be obtained first. Wouldn't it make more sense to have instead something like?

GET https://labs.j-novel.club/app/v1/series/{ID or slug}/parts

(ie using the series ID)

chocolatkey

@jhkghl678690iop I can add that. It was originally constructed this way for the app's sake

Malloc

I am having a hard time getting part data for a part that requires me to be logged in to download. On the old api, I would set up a header on the html GET request with a header called "Authorization" with a value of the sessionID I got from the login.

However, when I use the new API: "https://labs.j-novel.club/embed/{part ID}/data.xml" it always returns forbidden when I pass the session id in the Authorization header. I have the both the session id (from the old login API) and access cookie I got from the "set-cookie" header on the login response. However I don't know what header to put it in with the new api.

I have tried to figure out what headers to use however when downloading a part using your websites reader the headers appear to be hidden on the HTML request when using Chrome's developer tools but it appears the headers on the data.xml are hidden because it is embeded into a blob:https://labs.j-novel.club/XXXXX call

Crimson Wise

@malloc

I'm doing the following (replace XXX, YYY and ZZZ as required) and it works:

$ curl -X POST 'https://api.j-novel.club/api/users/login' -H 'Content-Type: application/json' -d '{"email":"XXX", "password":"YYY"}' -s
  {"id":"ZZZ","ttl":1209600,...}

$ curl 'https://labs.j-novel.club/embed/61d760a893e1a3e2123d0fb5/data.xhtml' -H 'Authorization: Bearer ZZZ' -s | head -n 6
  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE html>
  <html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xmlns:xml="http://www.w3.org/XML/1998/namespace" lang="en" xml:lang="en">
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="content-type"/>
    <title>Guide to the Perfect Otaku Girlfriend: Roomies and Romance Volume 5 Part 2</title>

You can also use instead the token from the new login endpoint and also works.

Malloc

@crimson-wise

Sweet. It was the additional "Bearer " I was missing in the authorization header. Thank you. I read about that in the first post so I thought they might have meant the authorization header was called "Bearer" and not that the authorization header was still authorization it just wanted a bearer in front of the token.

Thanks so much! I am gonna start working on replacing all of my old api calls with the new api ones one step at a time.

Malloc

said in Developer PSA: labs.j-novel.club is your new friend!:

Endpoints return protobuf-encoded responses by default. If you want to get JSON, add format=json to the query. If you want a text dump format, format=text.

Does anyone know where you put format=json? My application already deserializes json, and I was wondering how to request labs.j-novel.club to pass back the response in json.

I tried having a header format with a value of json and that still returns protobuf so I am obviously missing where I pass this in the query.

chocolatkey

@malloc it's a query parameter as it says. So in the URL query, e.g. /endpoint?format=json

Malloc

@chocolatkey Thanks again. I am relatively new to web development like this so you have been very helpful.

chocolatkey

@crimson-wise Don't forget you can use "slim": true in the login since you just need the token.

tim_pkmn89

I found out you guys had an API a few days ago, and thought I'd make use of it to play around with some new Python features.

A few questions/comments:

Is there a way to find out if a volume is still in the middle of releasing parts? Right now I'm just checking to see if the next volume exists. I don't want to make any assumptions based on the number of parts.
I noticed that the datetimes are different for the manga API calls. Specifically, the volume "publishing" field and the parts "expiration"/"launch" fields don't include milliseconds while everything else does. I had to add in some extra code to parse them out.
Ascendance of a Bookworm itself, V1, and V1P1 all share the same slug of "ascendance-of-a-bookworm". Is this known? It doesn't seem to be like that for any other title.

myskaros

@tim_pkmn89 said in Developer PSA: labs.j-novel.club is your new friend!:

Ascendance of a Bookworm itself, V1, and V1P1 all share the same slug of "ascendance-of-a-bookworm". Is this known? It doesn't seem to be like that for any other title.

I can answer this one; this is simply due to the way the original backend was programmed, where the slug is automatically generated based on the plain text title entered. Even if the title is changed afterward, the slug stays the same presumably since that slug is entered manually in other parts of the backend and it would be a pain to have to update all those fields if the slug were to change.

yanrishatum

Have to also note about /parts/$id/toc not making sense, as it clearly should be /series/$id/toc / /series/$id/parts.
Another thing I'd like to note is lack of parent IDs in models. When fetching /volumes/$id I'd expect it to have a seriesId field or similar, because sometimes I don't want to get all the data about series, just the ID to associate volume and series. Same applies to parts.
For example, to match old endpoint level of data just to show full novel information page the amount of requests is inadequate:

const series = await get(`/series/${id}`); // Basic series metadata
const volumes = await get(`/series/${id}/volumes`); // Volume list
const partsPerVolume = new Map(); // Part list
for (const volume in volumes) partsPerVolume.set(volume, await get(`/volumes/${volume.legacyId}/parts`));

We're looking at 2 + volume count requests. With /parts it can be reduced to constant 3 (and you still can do that in constant 5 by asking only first volume parts and getting toc afterward) requests, but there's a caveat: Parts don't currently reference their parent volume.
Of course we can do assumptions on slug generation and just parse it to extract volume or parse title, but it's an extremely dubious affair.

Now onto the data that is present in old API, but is lacking in new one:

Series:

creator metadata: Well, makes sense as it can change from volume to volume, so it's not a big deal.
status: It's not possible to know the current status (completely translated, hiatus, cancelled, etc)
totalVolumes: No way to know how many volumes series currently have, only way to know is to ask API to give us full list of volumes in a separate request.
availableVolumes: That one is a bit iffy, because in old API I've seen it being an unreliable information source.

Volumes:

JapanesePubDate: Doesn't provide info when COO volume was released. (However variable name could be better ;) )
approximatePageNumber for novels. It's fairly inconsequential, but was a nice statistic to see.
totalPartNumber: No way to know how many parts volume contains. I'd even say it needs both total and currently released part counts.
serieId: As stated before - only way to know parent series is to ask API, and it will give full series information, even if developer needs just an ID.

Parts:

volumeId: Same as before, only way to know is to ask API.
seriesId: Ditto.

And lastly, lack of relation expansion is hurting, even if by design, it should be cheaper both for user and server to just provide series info, it's volumes and parts in a singular request than to handle multiple requests. (But it needs to be on-demand, obviously)
I do have to say that new API is clearly better designed overall. But afformentioned issues with IDs and relation expansion inflate the request count far too much for no good reason.

Regarding ?format=json: Would be also helpful if server would respect Accept header. I.e. if it says application/json that means it expects JSON in return.