Developer PSA: labs.j-novel.club is your new friend!

redmasq

@chocolatkey Thanks for the hard work.

Concerning the username scraping, with that vector plugged up, the easiest way, if reading above correctly, would be once logged in, using rainbow tables, but a moot point since I'm fairly certain someone with a valid username can harvest from these forums anyways, in theory. Logging mixed with governor limits per login identity and per IP/subnet should mitigate much of that style. At least for stuff that I have maintained, an expiring minutely and a rolling daily counter set usually suffices. That selection allowed me to manage the minutely collection in-memory and queue up daily to allow atomic transactions for log and aggregate counts.

Regardless, hoping to setup keeping my local copies auto synced, so might code up a cron job this weekend. I may be willing to share a wrapper of whatever language I use if there is interest. Node.js (JavaScript), Python, or Powershell will be my most likely one of my choices, provided I don't get assigned honey-dos (not that this sort of thing is much more than http request and deserialization, but wouldn't be the first time).

chocolatkey

This post is deleted!

Crimson Wise

I am looking for some dogfooding

For reference, which kind of feedback are you looking for?

jhkghl678690iop

One thing is in order to get all the parts of the serie, it is possible to call:

GET https://labs.j-novel.club/app/v1/parts/{ID or slug}/toc

But the ID of a part of the series needs to be obtained first. Wouldn't it make more sense to have instead something like?

GET https://labs.j-novel.club/app/v1/series/{ID or slug}/parts

(ie using the series ID)

chocolatkey

@jhkghl678690iop I can add that. It was originally constructed this way for the app's sake

Malloc

I am having a hard time getting part data for a part that requires me to be logged in to download. On the old api, I would set up a header on the html GET request with a header called "Authorization" with a value of the sessionID I got from the login.

However, when I use the new API: "https://labs.j-novel.club/embed/{part ID}/data.xml" it always returns forbidden when I pass the session id in the Authorization header. I have the both the session id (from the old login API) and access cookie I got from the "set-cookie" header on the login response. However I don't know what header to put it in with the new api.

I have tried to figure out what headers to use however when downloading a part using your websites reader the headers appear to be hidden on the HTML request when using Chrome's developer tools but it appears the headers on the data.xml are hidden because it is embeded into a blob:https://labs.j-novel.club/XXXXX call

Crimson Wise

@malloc

I'm doing the following (replace XXX, YYY and ZZZ as required) and it works:

$ curl -X POST 'https://api.j-novel.club/api/users/login' -H 'Content-Type: application/json' -d '{"email":"XXX", "password":"YYY"}' -s
  {"id":"ZZZ","ttl":1209600,...}

$ curl 'https://labs.j-novel.club/embed/61d760a893e1a3e2123d0fb5/data.xhtml' -H 'Authorization: Bearer ZZZ' -s | head -n 6
  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE html>
  <html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xmlns:xml="http://www.w3.org/XML/1998/namespace" lang="en" xml:lang="en">
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="content-type"/>
    <title>Guide to the Perfect Otaku Girlfriend: Roomies and Romance Volume 5 Part 2</title>

You can also use instead the token from the new login endpoint and also works.

Malloc

@crimson-wise

Sweet. It was the additional "Bearer " I was missing in the authorization header. Thank you. I read about that in the first post so I thought they might have meant the authorization header was called "Bearer" and not that the authorization header was still authorization it just wanted a bearer in front of the token.

Thanks so much! I am gonna start working on replacing all of my old api calls with the new api ones one step at a time.

Malloc

said in Developer PSA: labs.j-novel.club is your new friend!:

Endpoints return protobuf-encoded responses by default. If you want to get JSON, add format=json to the query. If you want a text dump format, format=text.

Does anyone know where you put format=json? My application already deserializes json, and I was wondering how to request labs.j-novel.club to pass back the response in json.

I tried having a header format with a value of json and that still returns protobuf so I am obviously missing where I pass this in the query.

chocolatkey

@malloc it's a query parameter as it says. So in the URL query, e.g. /endpoint?format=json

Malloc

@chocolatkey Thanks again. I am relatively new to web development like this so you have been very helpful.

chocolatkey

@crimson-wise Don't forget you can use "slim": true in the login since you just need the token.

tim_pkmn89

I found out you guys had an API a few days ago, and thought I'd make use of it to play around with some new Python features.

A few questions/comments:

Is there a way to find out if a volume is still in the middle of releasing parts? Right now I'm just checking to see if the next volume exists. I don't want to make any assumptions based on the number of parts.
I noticed that the datetimes are different for the manga API calls. Specifically, the volume "publishing" field and the parts "expiration"/"launch" fields don't include milliseconds while everything else does. I had to add in some extra code to parse them out.
Ascendance of a Bookworm itself, V1, and V1P1 all share the same slug of "ascendance-of-a-bookworm". Is this known? It doesn't seem to be like that for any other title.

myskaros

@tim_pkmn89 said in Developer PSA: labs.j-novel.club is your new friend!:

Ascendance of a Bookworm itself, V1, and V1P1 all share the same slug of "ascendance-of-a-bookworm". Is this known? It doesn't seem to be like that for any other title.

I can answer this one; this is simply due to the way the original backend was programmed, where the slug is automatically generated based on the plain text title entered. Even if the title is changed afterward, the slug stays the same presumably since that slug is entered manually in other parts of the backend and it would be a pain to have to update all those fields if the slug were to change.

yanrishatum

Have to also note about /parts/$id/toc not making sense, as it clearly should be /series/$id/toc / /series/$id/parts.
Another thing I'd like to note is lack of parent IDs in models. When fetching /volumes/$id I'd expect it to have a seriesId field or similar, because sometimes I don't want to get all the data about series, just the ID to associate volume and series. Same applies to parts.
For example, to match old endpoint level of data just to show full novel information page the amount of requests is inadequate:

const series = await get(`/series/${id}`); // Basic series metadata
const volumes = await get(`/series/${id}/volumes`); // Volume list
const partsPerVolume = new Map(); // Part list
for (const volume in volumes) partsPerVolume.set(volume, await get(`/volumes/${volume.legacyId}/parts`));

We're looking at 2 + volume count requests. With /parts it can be reduced to constant 3 (and you still can do that in constant 5 by asking only first volume parts and getting toc afterward) requests, but there's a caveat: Parts don't currently reference their parent volume.
Of course we can do assumptions on slug generation and just parse it to extract volume or parse title, but it's an extremely dubious affair.

Now onto the data that is present in old API, but is lacking in new one:

Series:

creator metadata: Well, makes sense as it can change from volume to volume, so it's not a big deal.
status: It's not possible to know the current status (completely translated, hiatus, cancelled, etc)
totalVolumes: No way to know how many volumes series currently have, only way to know is to ask API to give us full list of volumes in a separate request.
availableVolumes: That one is a bit iffy, because in old API I've seen it being an unreliable information source.

Volumes:

JapanesePubDate: Doesn't provide info when COO volume was released. (However variable name could be better ;) )
approximatePageNumber for novels. It's fairly inconsequential, but was a nice statistic to see.
totalPartNumber: No way to know how many parts volume contains. I'd even say it needs both total and currently released part counts.
serieId: As stated before - only way to know parent series is to ask API, and it will give full series information, even if developer needs just an ID.

Parts:

volumeId: Same as before, only way to know is to ask API.
seriesId: Ditto.

And lastly, lack of relation expansion is hurting, even if by design, it should be cheaper both for user and server to just provide series info, it's volumes and parts in a singular request than to handle multiple requests. (But it needs to be on-demand, obviously)
I do have to say that new API is clearly better designed overall. But afformentioned issues with IDs and relation expansion inflate the request count far too much for no good reason.

Regarding ?format=json: Would be also helpful if server would respect Accept header. I.e. if it says application/json that means it expects JSON in return.

myskaros

@yanrishatum said in Developer PSA: labs.j-novel.club is your new friend!:

Now onto the data that is present in old API, but is lacking in new one:

I can't speak for chocolatkey, but as a sysadmin, most of that data you're saying is missing in the new API simply doesn't exist for a large majority of our series/volumes/parts. There were many miscellaneous data fields added over the years, but we never went back to ensure it was all filled out or check if it's all accurate. With development on the new backend underway, a lot of those data fields simply won't exist anymore, so that would explain why you can't obtain it in the new API.

I think many of your grumbles have more to do with how data is stored in our current backend more than how the new API functions. Which is to say, once the new backend and database are up and running, I believe things will be much more logical with the new API as well.

yanrishatum

@myskaros Totally fair, I don't care much about minor information, and main grumble is lack of parent IDs (I suppose those are stored in a separate relation tables as series<->volume / series<->part / part<->volume hence not listed, because that'd require extra queries?).
Didn't work databases much lately so I'm rusty, but I'm pretty sure automated update of volume/part counters can be done with relative ease.
Mainly I just don't like the idea of spamming server with multiple requests just to get full set of information to expand the relations, especially since it increases the latency at which said data can be shown (to be fair I can just show data as it comes in, and do a spinner for still loading parts, but still - far longer than one request that gets all the info) to user as well as just causes more strain on the server (one user is fine, but when there's lots of them - it adds up), and if mobile app would get more features such as more detailed novel information on a level of website - it can turn ugly.

chocolatkey

The old part data API is now "dead"

FalseAim

Can we get an endpoint for the calendar?

Alternatively, if the event data listed the id or slug of the associated part or volume, I could get the information that way (but with more calls). Currently the event data just lists data for the series, not the part or volume.

yanrishatum

@falseaim https://labs.j-novel.club/app/v1/events is what you're looking for. Take a look at how calendar currently operates. You can pass it sort, start_date (ISO string), end_date and skip arguments. Use launch sort and define start/end date to get pretty much exactly what calendar is displaying. And skip is obviously for pagination.
I.e. for example:
https://labs.j-novel.club/app/v1/events?sort=launch&start_date=2022-04-30T21%3A00%3A00.000Z&end_date=2022-05-31T21%3A00%3A00.000Z.
Each event provides valid IDs as well as series information and thumbnail.