#Separating Packages into Internal Subfolder

105 messages · Page 1 of 1 (latest)

lofty flicker
#

I have found that when organizing my projects I have lots of internal packages. These packages have different "layers" of usage where the main is the top layer and low level api are bottom layer and theirs interfaces in between. And my rule is that layers can only access packages below them.

I would like to be able to separate each layer into a separate internal folder (/internal/layer1 /internal/layer2) but I am not sure how to do this as the project structure seems to not lint or compile correctly when i use subfolders in the internal folder. (but these layers are all internal and not meant to be reused by external sources)

Is there any way to implement package subfolders in a go project?

(Also if anyone has any critique on how I setup my projects I would love to hear your opinion.)

Here is an example of my typical "layers" to better explain this:

lowest layer: External Packages, External API Implementations, and Databases ex: AWS s3, AWS SES, Sendgrid R2, Cloudflare Turnstile, Postgres, Sqlite, Turnstile.

2nd layer: Generalized Packages with interfaces that allow for redundancy and easily switching out providers/services, Email(api for sending email using multiple email providers), Translate(mutli provider support), Captcha(multi), Database(multi), FileStorage(multi).

3rd layer: Specific separate pieces of the app, Blog, ComplexLandingPage, Application, Authentication, Logging

top layer: package main: main.go and router

crystal pike
#

if layers can only access layers below them you can just have internal/highest_level/next_level.../lowest_level

#

or for the details you specified you can do something like internal/email_provider that will take care of all email providers and internal/database and same for all your services

#

for my backend projects I use something like

# 
main.go
internal/
  database/
  handlers/
  middleware/
  utils/
astral reef
#

It might help for you to share the lint error you are seeing, and a dummy project (with the folders and files) that replicates that exact error. You'll get more useful help that way, I think. if you're lucky, you might even solve your problem on the way of specifying your problem better.

chrome edge
crystal pike
chrome edge
#

it's convincing me to put it in blog form actually, as it keeps coming up

silver rune
#

Don’t do internal package until it’s really needed

#

n additional depths in package paths is pretty serious for what it gives

#

Hint; most people won’t care anyways about what you are about to make internal

chrome edge
lofty flicker
# chrome edge also don't overdo package hierachy (and for instance... you don't need internal/...

Yeah this is generally what i currently follow. I just have 2 folders currently

web/ main.go, setup.go, routers
internal/ all my actual packages

Honestly after thinking about this more and reading feedback I think that I am going to just stick with the 2 folder structure.

I'll try and minimize small individual packages and then use config structs(some name them model structs) and struct methods instead of normal functions for using external packages.

For example if my App has a Blog which needs Email which needs SES, i'll just use hierarchical structs:

AppConfig:{BlogConfig:{EmailConfig:{Providers:[SES]}}}}

and then use the methods of each struct field.

lofty flicker
lofty flicker
silver rune
#

I once wanted to add functionality to go toolchain, but there were internal packages everywhere and I gave up

#

I would have to fork go the language lmao

silver rune
#

And I see the point of making some functionalities ,,private”

#

But do I really need it or is it something in my head hugged

dull ibex
silver rune
#

Sometimes, yeah… however my point is that I think that there are too many cases of ,,internal” packages/unexported types being unexported just to be unexported (aka simply hidden) specifically in newbie developers’ code. Where instead imo it should be used as a tool to restrict access to unsafe and not useful alone apis. For example there is a reason I would not export http client in some api client struct - I don’t want users to tinker with transport layer. Or some tcp client where you never want to directly expose sink/reader

dull ibex
#

what if I need to tinker with the transport layer? what if I need to interact with the tcp client?

where's the seam that makes one ok but not the other?

silver rune
#

I think that transport layer is a "contract" between user and said service and it should generally not be changed

#

Which doesn't imply that there shall be no ways of changing utility stuff such as timeouts

dull ibex
silver rune
#

If users need to change it they either need different service or they need service to implement the other protocol

dull ibex
#

You claimed internal was bad because it gets in the way.
I understand that point, what I'm trying to understand now is how that relates to general unexported symbols, why is it ok for them to be unexported?

#

Because if you say "I always export everything, uncoditionally" I can understand that

#

If you say "I like to pick a choose what I export" then I want to understand where the seam is in the reasoning between packages and symbols

silver rune
#

I think that packages are generally a piece of code whose some features may be exported and some may not, if you choose to not export anything from a package why care to make it a package in 99% of situations

dull ibex
#

With respect to "the library getting in the way and requiring the user to have drastic measures, such as a fork"

dull ibex
#

because you said you do use unexported symbols - if you didn't then that would be it

silver rune
#

Most times I generally don't see reason to make such packages internal

dull ibex
#

which is a general statement

silver rune
#

It's totally okay to have unexported packages which doesn't mean you MUST have unexported packages in your codebase

#

There is a blurry line between usefulness and security/safety/compatibility/etc and what I said is that people should think again before making package internal and maybe more generally before unexporting

#

And that imo unexporting stuff should be a result of security/safety/compatibility/etc decision and not just to do it because convention on github/someone said so (or because you think it's not useful anywhere else besides the code that'd consume the unexported type)

dull ibex
#

people should think again before making package internal
Because it might be useful for someone else, correct?
Doesn't that imply that you now have to think about this hypothetical someone else and design to a much higher standard than you would need to if you were the sole consumer?

Don’t do internal package until it’s really needed
When it is needed it's already too late. The reasoning behind the advice on https://go.dev/doc/modules/layout is precisely that it is cheaper to expose something previously unexposed than to unexpose something that previously was.

And if we bring the "why would you use internal if no one will import it anyway" then why would you pay the cost of designing to a standard where it would be imported if no one ever will?

silver rune
# dull ibex > people should think again before making package internal Because it might be u...

  1. I'd just export everything as long as I think it's safe and unexport something that can be potentially used illegally in our api, such as for example initialization calls

  2. Maybe that was poorly worded, until in this context meant "don't use internal package before you have a reason to do so" (rethink that decision).

And if we bring the "why would you use internal if no one will import it anyway" then why would you pay the cost of designing to a standard where it would be imported if no one ever will?
What does one have to do with another.
Is making apis used by main an internal a standard?

dull ibex
# silver rune 1. I'd just export everything as long as I think it's safe and unexport somethin...

  1. suffers from the exact same problems as your original "internal gets in the way" statement. Just because it can be used illegally it doesn't mean that it will, and you are getting in the way of users that need to cope with the api you have provided and are now unable to because you have unexported symbols.

  2. if you mean "don't use it without a reason to" then we're in agreement, that should indeed go without saying

What does one have to do with another.
It follows from my first inference:
Because it might be useful for someone else, correct?
Doesn't that imply that you now have to think about this hypothetical someone else and design to a much higher standard than you would need to if you were the sole consumer?

Is making apis used by main an internal a standard?
The "standard" (I heavily dislike that word and it's a shame it's still used) proposed in https://go.dev/doc/modules/layout is that if you have a supporting package the default choice should be for it to be unexported - until it is proven that it needs to be exposed

#

Much in the same way that if you create a type/variable/field/function/method, it should be unexported until you have a reason to export it.

#

This isn't a universal opinion. There is a "faction" for lack of a better word of our community that believes everything should be exposed, unconditionally, precisely because when it isn't it will get in the way of someone. That to me is a perfectly logical argument. It's not without faults, much like the opposing argument isn't, but it is consistent.

#

And by community I don't mean go exclusively

chrome edge
#

Not everything should be exposed, that's what lowercase functions and types are for... yet for most people internal/ is more smell than anything

dull ibex
#

That's just a statement, not an argument

#

If internal is a smell because it gets in the way (if, I'm trying to derive what your argument was supposed to be so do correct me if I'm wrong) then so do unexported symbols

chrome edge
#

the first reply to this long in the tooth thread is a perfect example of very bad layout

main.go
internal/
  database/
  handlers/
  middleware/
  utils/

(double whammy of internal/ and internal/utils/)

dull ibex
#

that is again just another statement

chrome edge
#

the argument is keep things simple, write code, don't "layout" according to whatever you've read before you have something useful and working; the other part of the argument is nobody is going to reuse what most people write... so write code for yourself first, with a good yet simple structure

dull ibex
#

if no one is going to re-use it, why should it be re-usable?

chrome edge
#

it doesn't need to be hidden either... why hide something, have an extra directory in every single import line...

dull ibex
#

then how does that correlate with

Not everything should be exposed, that's what lowercase functions and types are for...

#

that is the whole point I am trying to understand, where is the seam that makes unexported symbols ok and unexported packages a "smell"

#

I cannot tell you a single time in the 10 years I've been doing go that I cared about an import line

chrome edge
#

differentiating between implementation detail and usable api (even if used only internally) is just good coding... shoving it into internal/ so it can't escape is paranoia and over complication (in most cases... obviously if you have a complex api used by thousands, it makes more sense)

dull ibex
#

why is symbol encapsulation "good coding" even when there are 0 consumers and package encapsulation "bad coding" unless there are thousands of consumers?

#

the exact numbers are obviously meaningless, I'm just going with what you said

chrome edge
#

there aren't 0 consumers, there is the local consumer at least... (unless it's a weird "build it and they'll come" which doesn't work for libraries, you need at least 1 (probably 3) things using a library for it to make sense to exist)

dull ibex
#

that would also apply to an internal package, unless I'm missing a broader point

chrome edge
#

if you agree that go's stdlib isn't the same as joe's little project in term of breadth of content and usability and commitments to maintaining things "forever" then hopefully you can agree that yes there is a difference/a cutting point some where in the middle and that cutting point is not "joe's first project"

dull ibex
#

but why is it good coding for joe to use symbol encapsulation and not package encapsulation?
because that's what internal is, lowercase for packages

chrome edge
#

if you're otel or k8s... use internal/ if you're not/just starting, don't... and stay at 0.x

#

because you don't need package encapsulation

dull ibex
#

then where's the seam? why does joe need one and not the other?
is the premise that since no one will import it internal won't do anything?

chrome edge
#

give me a concrete example... and we can discuss whether it's useful or not

#

it's like the argument about utils/ it's an easy yet bad name to pick

dull ibex
#

I'm not the one making blanket statements, I don't see why I should be the one to conjure a concrete example

chrome edge
#

I'm not making a blanket statement at all... I'm saying most projects don't need it and some do, yet because of copying what they see in other projects most people seem to put themselves in the wrong bucket.

dull ibex
chrome edge
#

if anything you're the one making the blanket statement that everyone should use internal/

dull ibex
#

I never said anything remotely close to that

chrome edge
#

and I'm asking why most projects don't need it
Same as everything else in software: if you don't actually need it, don't use/have it... simplest that works but not simpler is the northstar

dull ibex
#

but that doesn't answer why I don't need it

#

that is my question, why don't I need it?

#

where "I" is your joe

chrome edge
#

if you (joe) ask yourself (themselves) what purpose does it serve, and you have a valid answer, then go ahead... I'll have done my job (unlike that doc post which I think could use refinements... though if you see 1 below the case with internal and single package, they have multi where they use the same files not in internal anymore)

dull ibex
#

it serves the (almost) exact same purpose as symbol encapsulation does, which we established is "good coding" even for joe

#

the main distinction between both types of encapsulation is that symbol encapsulation plays a role in guaranteeing invariants, whereas package encapsulation (hopefully) does not

#

the rest of the circle for both is the same: exporting something is a commitment to supporting it

#

so, why does your example joe not need package encapsulation? as you have stated.
is it because no one will import joe's packages?

#

is it because it's an extra path segment in the import line?
is it because it's an extra directory?

#

that's what I'm trying to get to the bottom of, trying to understand the actual argument behind the statements

chrome edge
#

is it because no one will import joe's packages?
they surely won't if it's all tucked in internal/ but

just have joe use their own stuff first and maybe it'll evolve into something useful after dozens of versions/iterations (hopefully all in 0.x) and then they can decide what is the minimal api surface

#

and yes it's also marginally because of the the little extra too

#

which will slow joe down vs quick iterations

dull ibex
#

they surely won't if it's all tucked in internal/
won't what? I'm not seeing what this is in response to

#

is it "need package encapsulation"?

chrome edge
#

it's in reply to

is it because no one will import joe's packages?

dull ibex
#

so if the argument is no one will import it anyway then why should it be exposed?
we could apply that exact same reasoning to general symbols, if no one will use them why should they be hidden?

just have joe use their own stuff first and maybe it'll evolve into something useful after dozens of versions/iterations (hopefully all in 0.x) and then they can decide what is the minimal api surface
that's precisely the reasoning behind the advice in https://go.dev/doc/modules/layout
starting out as unexposed gives you maximum freedom to iterate, literally, you can't do better than that

on the other hand, versioning applies wholesale to the whole module, not to individual packages
I cannot say package a is 1.x and package b is 0.x

#

but if package b is not exposed, than it is 0.x by definition, an enforceable definition

#

so if the argument is no one will import it anyway then why should it be exposed?
but this is really the crux of the matter

if you have a public api you have the responsibility to design it to a much higher standard than you'd need to if it were private
precisely because if it is private you maximum freedom to iterate and change your mind

if an api is not meant to be public, "no one will import it anyway" is in my opinion a bad reason to not put it in internal
if it weren't, then we'd do the same for general symbol visibility, and we don't. According to you it is "good coding" to keep private things private.
so if my package is for all intents and purposes private, why should I not keep it private?

That's the second stage of the argument, where joe already knows it's a private package.
As for the first stage of the argument, where joe does not yet know it's a private package then defaulting to private allows joe to keep ALL of his options open. He can rewrite it 30x and no one will care, because no one is using it. He can throw it away. He can decide to move it to a "staging" period in some exp/ like thing. Joe has maximum freedom to iterate by defaulting to not immediately exposing the new package.

chrome edge
#

well ... I disagree, it's premature structuring (as a strawman why not make the repo private too, that avoids it being misused)

#

also while I think agile is mostly to sell books and consulting... it's not "agile"

dull ibex
#

why is it premature, at what point would it be acceptable to effectively delete a public package by making it internal?

#

And if you mean it as "it's premature to assume someone will use it" then why does that not apply to symbol visibility?
If there's no reason to believe someone will use it, why would you make it usable?
These are still open questions, or at least I've not understood their rebuttal.

Do we not already accept that it's usually a good idea to enforce your expectations? Is that premise not prevalent in our everyday lives, by not exporting sensitive symbols, by using type systems, by guaranteeing invariants?

Of course, there's no such thing as a free lunch, not exporting a symbol has a lower cost to the maintaners than introducing a new directory.
Going all out on type safety lands you in the lands of rust and haskell, which we as go programmers seem to view as more complex when compared to go, a major tradeof.

So, what's the cost of internal beyond introducing a directory?
Is the cost of a directory not worth guaranteeing your assumption that no one will use it?
If that's your assumption does that not contradict previous points that assuming such is harmful to the users as someone may want too?
If you expose something, with the assumption that no one will use it, but force yourself to make it usable in the fear of getting in the way of someone that you assume does not exist, are you not burdening yourself by elevating the standard of design required thereby affecting, in a measurable way, your speed of iteration?