How to create multiple XML files if single XML file matches the requirements?

Below is my client code which stream all the customer url's from golang grpc server and it works fine. It takes Request input parameter and streams customer url's basis on a particular clientId. In my below code, I am streaming all customer url's for ClientId 12345 and it works fine.

I am also creating an XML file with all the URL's in it for particular clientId as shown below. For example: Below will create 12345_abc.xml XML file with all the URL's in them in particular format.

func main() {
// this "clientId" will be configurable in future
clientId := 12345
timeout := time.Duration(1000) * time.Millisecond
ctx, _ := context.WithTimeout(context.Background(), timeout)
conn, err := grpc.DialContext(ctx, "localhost:50005", grpc.WithInsecure())
if err != nil {
log.Fatalf("can not connect with server %v", err)
}
// create stream
client := pb.NewCustomerServiceClient(conn)
req := &pb.Request{ClientId: clientId}
stream, err := client.FetchResponse(context.Background(), req)
if err != nil {
log.Fatalf("open stream error %v", err)
}
// create new object to populate all URL data in memory
urlHolder := NewClient()
t := time.Unix(0, 0).UTC()
done := make(chan bool)
go func() {
for {
resp, err := stream.Recv()
if err == io.EOF {
done <- true
return
}
if err != nil {
log.Fatalf("can not receive %v", err)
}
log.Printf("Resp received: %s", resp.GetCustomerUrl())
// populate URL object with all the required field in it
urlHolder.Add(&URL{
Loc:        resp.GetCustomerUrl(),
LastMod:    &t,
ChangeFreq: Daily,
Priority:   10.2,
})
}
}()
<-done
log.Printf("finished")
// create an XML file with all the URL's in it and then save it on disk
// for particular clientId. This will create "12345_abc.xml"
file, _ := os.Create(fmt.Sprintf("%d_abc.xml", clientId))
urlHolder.WriteTo(file)
}

Here is my urlholder.go file:

type URL struct {
Loc        string     `xml:"loc"`
LastMod    *time.Time `xml:"lastmod"`
ChangeFreq ChangeFreq `xml:"changefreq"`
Priority   float32    `xml:"priority"`
}
type UrlMap struct {
XMLName xml.Name `xml:"urlset"`
Xmlns   string   `xml:"xmlns,attr"`
URLs    []*URL   `xml:"url"`
Minify  bool     `xml:"-"`
}
func NewClient() *UrlMap {
return &UrlMap{
Xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9",
URLs:  make([]*URL, 0),
}
}
func (s *UrlMap) Add(u *URL) {
s.URLs = append(s.URLs, u)
}
// WriteTo writes XML encoded urlMap to given io.Writer.
func (s *UrlMap) WriteTo(w io.Writer) (n int64, err error) {
cw := NewCounterWriter(w)
_, err = cw.Write([]byte(xml.Header))
if err != nil {
return cw.Count(), err
}
en := xml.NewEncoder(cw)
if !s.Minify {
en.Indent("", "  ")
}
err = en.Encode(s)
cw.Write([]byte{'\n'})
return cw.Count(), err
}

Here is my CounterWriter class -

// CounterWriter implements io.Writer. Count of bytes written is tracked.
type CounterWriter struct {
writer io.Writer
count  int64
}
var _ io.Writer = (*CounterWriter)(nil)
// NewCounterWriter wraps io.Writer and returns CounterWriter.
func NewCounterWriter(w io.Writer) (cw *CounterWriter) {
return &CounterWriter{
writer: w,
}
}
// Write calls Write on the wrapped io.Writer and adds the number of bytes
// written to the counter.
func (cw *CounterWriter) Write(p []byte) (n int, err error) {
n, err = cw.writer.Write(p)
cw.count = cw.count + int64(n)
return n, err
}
// Count returns the number of bytes written to the Writer.
func (cw *CounterWriter) Count() (n int64) {
return cw.count
}

Problem Statement

Above code works fine but I need to split an XML file into multiple XML files for same clientId if it matches below requirements:

• A single XML file should not be more than 50MB max. It can be approximate, doesn't have to be accurate.
• A single XML file should not have more than 50K URL's max.

I know it's weird that 50k URL limit will be reached sooner than 50MB limit but this is what I got the requirement. Now basis on above logic, I need to make multiple XML files for particular clientId. All those multiple files can be like this 12345_abc_1.xml, 12345_abc_2.xml or any other better naming format. I am kinda confuse on how should I proceed to do this.

I can add logic for 50K url by using for loop but confuse on the size logic and also I want to make this generic for each clientId so I am having difficulties doing this.

Inside your WriteTo function, you should be calling something like w.Write(myBytes).

The size of myBytes inside that function is the size that you are looking for. You can get it using len(myBytes) or with the first return of w.Write(myBytes). This is important because there is no way of "estimating" the size that a file would have, other than directly counting the information that you will write.

You are converting UrlMap into bytes somewhere inside your WriteTo function. That means you can do the same with any URL variable.

The way that I would solve this problem is to have a sizeCounter and add the number of bytes that would be stored everytime I create a new URL variable inside the for { loop. In the same place I would also count the number of URLs created. With both counters then the rest is easy.

I would add the transformation from URL to bytes inside the .Add function and return it so that everything is easier to understand. You are going to have to move some variables into the go routine.

func (s *UrlMap) Add(u *URL) (int) { // Modify this function to count the size and return it
    s.URLs = append(s.URLs, u)

    var urlBytes []byte
    var err error

    urlBytes, err = xml.Marshal(u) // Transform to bytes using xml.Marshal or xml.MarshalIndent
    if err != nil {
        panic(err) // or return the error if you want
    }

    return len(urlBytes)
}

    t := time.Unix(0, 0).UTC()
    done := make(chan bool)
    go func() {
        // create new object to populate all URL data in memory
        urlHolder := NewClient()
        urlCounter := 0
        byteCounter := 0
        fileCounter := 0

        for {
            resp, err := stream.Recv()
            if err == io.EOF {
                done <- true
                file, _ := os.Create(fmt.Sprintf("%d_abc_%d.xml", clientId, fileCounter))
                urlHolder.WriteTo(file)
                return
            }
            if err != nil {
                log.Fatalf("can not receive %v", err)
            }
            log.Printf("Resp received: %s", resp.GetCustomerUrl())
            // I add the bytes of the URL here as a return
            urlBytes := urlHolder.Add(&URL{
               Loc:        resp.GetCustomerUrl(),
               LastMod:    &t,
               ChangeFreq: Daily,
               Priority:   10.2,
            })
            byteCounter += urlBytes
            urlCounter += 1
            if byteCounter > 49000000 || urlCounter >= 50000 { 
                file, _ := os.Create(fmt.Sprintf("%d_abc_%d.xml", clientId, fileCounter))
                urlHolder.WriteTo(file)
                urlHolder = NewClient() // create a new object for next loop
                fileCounter += 1 // prepare fileCounter for next loop
                byteCounter = 0 // restart count variables
                urlCounter = 0
            }
        }
    }()

    <-done
    log.Printf("finished")
    // No longer write the files here.